Purpose

Adds vllm analyze-prefix-cache: an offline, no-GPU CLI that estimates how prefix-cache-friendly a request dataset is, before spending GPU time on a serving run.

Fixes #47993.

Motivation: vLLM's automatic prefix caching reuses KV-cache blocks only when the full-block hash chain matches exactly. Small prompt-formatting differences, per-request metadata, or chat-template variance can silently prevent reuse. Today there's no way to answer "is my RAG/agent/multi-turn dataset cache-friendly?" without actually running inference. This CLI answers that statically: tokenize the dataset, compute the same full-block hash chain the V1 scheduler uses, and report cacheability.

$ vllm analyze-prefix-cache --model meta-llama/Meta-Llama-3-8B-Instruct \
    --input requests.jsonl --block-size 16

requests analyzed        : 500
block size                : 16
total prompt tokens       : 184320
total full-block tokens   : 179200
est. reusable full blocks : 61440
est. cacheability ratio   : 34.29%

top shared-prefix groups (offline upper bound, not a hit-rate guarantee):
  1. 128 requests share 512 full-block tokens (32 blocks) -> req_003, req_017, req_022, req_041, req_058 ...
  ...

v1 scope (per the design discussion in #47993, and my scoping comment there): plain-prompt JSONL only — OpenAI chat/batch JSONL is a fast-follow once this report shape is agreed on. No LoRA/multimodal/cache-salt extra-key support yet. Reuses hash_block_tokens / get_hash_fn_by_name / init_none_hash from vllm/v1/core/kv_cache_utils.py directly rather than adding a new public helper — the RFC flagged this as an open layering question; happy to help design a stable public helper afterward once there's a second caller to generalize from.

Design note: a real bug I found and fixed while building this

The most important number this tool reports — estimated_reusable_full_block_tokens (and the cacheability ratio derived from it) — was double-counting in my first implementation. Writing this up because it's the one non-obvious design decision in the diff, and I want it visible for review rather than buried.

The bug: requests are grouped by their longest shared hash-chain prefix, and I originally summed len(prefix) * block_size * (num_requests - 1) across the top-K reported groups. That's wrong whenever two distinct-membership groups overlap in blocks — which happens whenever a subset of requests shares a longer prefix than the full set does. Example:

a: [h0, h1, h2, h3, h4, hA_only]
b: [h0, h1, h2, h3, h4, hB_only]
c: [h0, h1, hC_diverge]

{a,b} share a 5-block prefix; {a,b,c} share a shorter 2-block prefix within that same lineage. Both are real, legitimately-reported groups. Summing them independently double-counts blocks 0-1, which belong to both:

buggy:   5*16*(2-1)  +  2*16*(3-1)  =  80 + 64  = 144
correct (verified by hand-walking the trie):            = 112

The fix: block hashes are already chained on their parent (hash_block_tokens(hash_fn, parent, tokens)), so a given BlockHash value uniquely identifies one trie node across every chain that reaches it — regardless of which top-K group it happened to land in. Counting reuse once per distinct BlockHash value, not once per reported group, gives the correct non-overlapping total:

block_hash_counts: dict[BlockHash, int] = defaultdict(int)
for chain in chains.values():
    for block_hash in chain:
        block_hash_counts[block_hash] += 1
reusable_tokens = sum(
    block_size * (count - 1) for count in block_hash_counts.values() if count > 1
)

Extracted into _reusable_tokens_from_chains() so it's unit-testable independent of the tokenizer (see Test Plan). I verified the fix against two hand-computed cases (a 3-way/2-way overlap, and a 3-way-then-2-way case) with a standalone script before writing it into the module — both matched the manual trie walk exactly (112 and 80 respectively).

The top_prefix_groups display logic is unaffected by this — it's fine for both {a,b}@depth-5 and {a,b,c}@depth-2 to be reported as separate groups in the human-readable output, since they're genuinely different sharing clusters. Only the aggregate reusable-token total needed to stop double-counting the overlap.

Changes

• vllm/entrypoints/prefix_cache_analysis.py (new): JSONL parsing, full-block hash-chain construction (mirrors hash_block_tokens block-by-block, dropping partial trailing blocks), prefix grouping for display, and the corrected _reusable_tokens_from_chains accounting.
• vllm/entrypoints/cli/analyze_prefix_cache.py (new): AnalyzePrefixCacheSubcommand, following the same CLISubcommand pattern as collect_env.py — --model, --input, --block-size, --hash-algo, --output-format {text,json}, --top-k-groups, --trust-remote-code.
• vllm/entrypoints/cli/main.py: registers the new subcommand (2-line addition, same pattern as every other CLI subcommand).
• tests/entrypoints/unit_tests/test_prefix_cache_analysis.py (new): JSONL parsing (happy path + missing-field error), end-to-end grouping/divergence via the real tokenizer (facebook/opt-125m), JSON round-trip, zero-full-blocks edge case, and a dedicated regression test for the double-counting bug above using synthetic BlockHash values (no tokenizer dependency, so it can't be broken by tokenizer/BPE changes).

Non-goals for v1

Matches the RFC: no runtime hit-rate prediction, no scheduler/eviction/preemption modeling, no natural-language miss classification, no LoRA/multimodal/cache-salt inference, no engine behavior changes. This is a static, offline upper-bound estimate — the text output says so explicitly ("not a hit-rate guarantee").

Test Plan

# New tests
pytest tests/entrypoints/unit_tests/test_prefix_cache_analysis.py -v

# Lint (run against the actual module paths, not extracted copies)
ruff check vllm/entrypoints/prefix_cache_analysis.py \
  vllm/entrypoints/cli/analyze_prefix_cache.py \
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py
ruff format --check <same files>

# Manual CLI smoke test
vllm analyze-prefix-cache --model facebook/opt-125m --input requests.jsonl --output-format json

Test Result

• ruff check and ruff format --check: clean.

• Every import (vllm.tokenizers.get_tokenizer, vllm.utils.hashing.get_hash_fn_by_name, vllm.v1.core.kv_cache_utils.{BlockHash, hash_block_tokens, init_none_hash}) checked against current main signatures directly, not from memory.

• The reusable-token fix verified independently with a standalone script against two hand-computed synthetic cases before being written into the module (see Design note above) — both matched a manual trie walk exactly.

• pytest actually run, on a throwaway Linux x86_64 CPU box (this repo's dev machine is macOS/arm64, which vLLM doesn't ship precompiled wheels for — VLLM_USE_PRECOMPILED=1 uv pip install -e . fails there with "No precompiled vllm wheel found for architecture arm64"). Fresh git clone of this branch, fresh uv venv, VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto, then:

  $ python -m pytest --noconftest tests/entrypoints/unit_tests/test_prefix_cache_analysis.py -v
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py::test_load_plain_prompt_jsonl PASSED
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py::test_load_plain_prompt_jsonl_missing_field PASSED
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py::test_analyze_groups_shared_prefix_and_splits_on_divergence PASSED
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py::test_analyze_report_json_roundtrip PASSED
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py::test_analyze_no_full_blocks_reports_zero_cacheability PASSED
  tests/entrypoints/unit_tests/test_prefix_cache_analysis.py::test_reusable_tokens_does_not_double_count_overlapping_groups PASSED
  ======================== 6 passed, 2 warnings in 12.98s ========================
  

  (--noconftest because vLLM's shared tests/conftest.py pulls in tblib and other heavy test-suite-wide fixtures this self-contained test doesn't need — installing the full requirements/test/*.in set felt disproportionate for one file with no shared fixtures.)

  First run of this actually caught a real bug — in the test, not the implementation: test_load_plain_prompt_jsonl asserted the wrong default id for a record following a blank line (expected "1", got "2"). load_plain_prompt_jsonl's own docstring says the default id is the 0-indexed physical line number, which correctly counts the skipped blank line — so "2" was right and the test's expectation was wrong. Fixed the assertion, not the code. Second run: 6/6 green.

AI assistance disclosure

Claude Code assisted with implementation, caught the double-counting bug described above during review of my own first draft, and caught the test-assertion bug during the actual Linux test run; commits are marked Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>. I reviewed and understand every line, including the trie-accounting argument above, and can defend it in review.